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Executive  Suminary 

Value  Added  Analysis  (VAA)  is  the  ^ativLin  th^ 

.<iecisiomnakers  in  evaluating  and  ^,^esl  VAA  helps  to  evaluate  the 

Program  Objective  Memorandum  interest  to  force  level  performance, 

relative  contribution  ("Value  a  wide  range  of  program  issues. 

VAA  is  fully  capable  Air  Defense  Anti-Tank 

Recent  systems  that  have  been  evaluated  us  g  (rAH-66),  and  the  Line  of 

System  (ADATS),  Reconn^ssance  and  A  j^^th^ology  is  expanding  beyond  the 

Sight  Anti-Tank  appropriations  and  growing  in  popularity 

research,  development,  and  ^  I  -^e  purpose  of  this  report  is  to  provide  a- 

-modifications  to  the  methodology.- 

VAA  is  a  useful  .00,  .0  use  in  P"""—  - 

■effectiveness.  It  provides  an  ‘"*'8^'  °  ^  obligation  Authority  (TOA).  It  can 

feasible  to  buy  when  changes  ate  mad  . .  j.  svstems  no  longer  recommended, 
force  '’wTou^ut  indicates  the  types  and  quantities  of  systems 

indicaung  possible  ^^^de-of  .  maximum  effectiveness  at  any  given  TOA 

of  interest  “  ^  include  a  funding  and  acquisition  stream  which  can 

rSi^plarg^Vammingthesystemsofintem. 

several  majorissuesimpacttheverificanono^^^^^^^^^ 

involves  the  measnres  of  “Imparinfc^^^^^  scaling,  and 

methods^  The  issues  concemi  g  anaivric  framework  used  to  prioritize  among 

weighting  them.  The  ^3  blLd  upon  the  analytic  hierarchy  process 

competing  altemanves.  The  possibility  of  rank  reversal,  the  frame  of 

(AHP).  The  issues  concerning  the  AR?  •  P  Technique  for  Order 

reference, and  the  scale  (TOPSIS)  algorithm  to  combine  different 

Preference  by  Similanty  nature  to  those  concerning  AHP, 

MOEs.  The  issues  the  form  of  rank  ordered  data  coefficients 

with  one  exception.  ^  ,  rnefficients  are  based  on  a  simple  rank 

based  on  an  oidinal  scale.  This  ^  an  ordinal  scale  can 


.MOBsamed.ria„hichespmss.heex,en,mwhichasys.en.pedonnsa«si<assigned.o^ 

system  under  a  specified  set  of  conditions. 
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scale,  which  is  distance  preserving.  The  coefficients  represent  the  relative  benefit  of 
competing  systems  and  must  express  the  relationship  between  the  variables  being 
compared.  They  also  need  to  satisfy  the  requirements  for  a  linear  program  model  i.e, 
they  must  satisfy  the  properties  of  certainty  and  linearity^. 

The  proposed  solution  to  the  issues  is  based  on  synthesis  of  the  Analytic  Hierarchy 
Process  (AHP)  assessment  methodologies  with  Multi-Attribute  Utility  Theory  (MAUT). 
The  solution  involves:  modeling  the  preferences  curves  of  decisionmakers,  plotting 
actual  combat  and  non-combat  results  on  these  curves  to  obtain  their  corresponding 
utility  value,  multiplying  the  utility  values  by  weights  obtained  using  the  AHP,  and 
using  an  additive  measurable  value  function  to  combine  effectiveness  values.  This 
eliminates  the  need  for  the  TOPSIS  algorithm  and  properly  scales  the  coefficients  for  use 
in  the  objective  function. 

This  report  presents  this  issue  and  others  and  provides  key  recommendations.  The 
proposed  recommendations  should  make  VAA  an  even  better  decision  analysis 
framework  for  including  or  excluding  various  systems  or  mixes  of  systems  from  the 
modernization  program. 


^Cook,  Thomas  M., Introduction  to  Management  Science,  4th  edition  ,  Prentice  Hall,  New  Jersey, 
1989,p50-51. 
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1.  Overview 

The  report  is  comprised  of  two  main  sections:  a  validation  section  ^d  a  verificatton 

'Sectiom  The  validation  section  shows  the  process  of  determining  the  ^tent.towhic  e 

overall  VAA  methodology  accurately  represents  the  real-world  from  the  perspective  o 

its  intended  use.  The  validation  process  is  ongoing  and  is  performed  dunng  each  stage  of 

the  methodology  in  conjunction  with  the  verification  process.  The 

shows  the  process  of  determining  if  the  models  and  simulations  used  in  the  methodology 

accurately  represent  the  Concepts  Analysis  Agency's  (CA  A)  concepmal  ^ 

specificaLns[l].  The  verification  process  will  exarmne  stated  and  implied  assumptions, 

relationships  within  and  between  different  levels  of  the  methc^ology, 

models  and  simulation  logic  correctly  perform  their  intended  functions,  reiaew  m^el 

inputs  and  outputs,  and  check  for  unexpected  sensitivity  or  lack  of  sensitivity  to  key 

inputs. 

There  are  eight  modules  in  the  VAA  standardized  approach  (see  figure  1.1).  The 
modules  are:  issue  clarification,  cost,  explicit  effectiveness,  implicit  effectiveness, 
effectiveness  integration,  optimization,  resource  allocatiori,  and 

report  will  focus  on  the  first  six  modules.  The  resource  allocation  module  and  the  results 
and  display  module  were  not  developed  for  the  prototype  VAA  study  and  will  not  be 
evaluat^.^The  resource  allocation  module  is  included  in  figure  1.1  for  display  purposes. 


Figure  1.1.  Value  Added  Analysis  Methodology  Overview 
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Each  of  the  modules  are  interrelated.  The  issue  clarification  module  consists  of  defining 
and  understanding  the  elements  of  the  tradeoff  issue.  The  cost  module  calculates  the 
costs  associated  with  each  alternative  for  the  issue  being  investigated.  The  explicit 
effectiveness  module  measures  the  quantitative  components  of  value  and  analyzes  the 
contribution  of  the  systems  of  interest  to  force  level  performance.  The  implicit 
effectiveness  module  measures  the  qualitative  components  of  value  and  analyzes  hard  to 
quantify  factors  such  as  political  risk,  program  flexibility,  and  contribution  to  industrial 
preparedness  as  well  as  other  criteria.  The  effectiveness  integration  module  uses  the 
Technique  for  Order  Preference  by  Similarity  to  Ideal  Solution  (TOPSIS)  algorithm  to 
combine  the  explicit  and  implicit  measures  of  value.  The  rank  ordered  data  coefficients 
from  TOPSIS  are  used  to  form  the  objective  function  coefficient,  vij,  which  is  the  per- 
item  contribution  of  the  system  to  force  effectiveness.  The  optimization  module  takes  vij 
and  maximizes  it  with  the  quantity  of  that  system  procured  in  a  particular  year  (xjj).  The 

maximization  is  then  subjected  to  budgetary,  force  structure,  and  production  constraints 
using  a  multiobjective  linear  program  called  the  force  modernization  analyzer 
(FOMOA). 
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2.  Validation 

This  section  indicates  the  extent  to  which  the  overfl  methodology  accurately  tepmsents 
the  real-world  from  the  perspective  of  its  intended  use. 

2.1.  Architecture 

Hacho,themod„.suse^^^^ 

liie  cycle  weapo  :  _  ^  ^  »vr.iirit  pffrctiveness  module  measures  an  issue  s  combat 

in  procurement  quantities.  The  explicit  effectiveness  moamc  ^^e  implicit 

integranon  muuuit  V.  tVip  nntnut  from  TOPSIS  is  in  the  form  of 

S:":::  rcSSrMmizIL  mV  maximizes  the  coefficients  and 
Strategy  based  on  cost-effectiveness  (see  figure  2.1.) . 


Module  Anaivtig  Memal 

Issue  Clarification  Information  Matrix 

Costing  Cost  Quantity  Model 

Life  Cycle  Cost  Model 

Explicit  Effectiveness  CORBAN 

Implicit  EHectiveness  Saatys  Pairwise 

Comparison  Technique 

Hfectiveness  TOPSIS 

integration 

Optimization  FOMOA 


Figure  2.1.  Overall  Meihodology  and  Analytic  Methods 
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2.2.  Input 

Issues  to  be  considered  are  decided  jointly  by  project  sponsors  and  VAA  analysts.  The 
selection  of  issues  is  based  on  the  importance  of  the  issue  to  the  decision  making  process 
and  the  ability  to  do  a  credible  job  modelling  the  systems  involved.  The  entire  problem 
is  framed  by  the  selection  of  the  systems  to  be  considered  and  by  the  year/scenario  of  the 
combinations  to  be  played[2]. 

2.3.  Procedures 

The  MOEs  of  an  alternative  are  obtained  using  two  different  methods.  The  first  method 
uses  CORBAN  to  obtain  the  combat  MOEs.  CORBAN  is  a  corps-scope,  battalion 
resolution  combat  simulation.  MOEs  produced  by  CORBAN  are  battlefield  snapshots, 
decision  traces,  and  atuition  statistics.  Each  battlefield  snapshot  indicates  the  locations, 
status,  and  current  activities  of  all  units  in  the  simulated  battle  at  a  particular  time.  A 
decision  trace  records  significant  decisions  made  by  all  units  over  a  panicular  interval  in 
the  battle.  Attrition  statistics  indicate  the  particular  types  of  assets,  and  the  quantities 
destroyed  by  opposing  weapon  types.  The  second  method  uses  two  surveys  to  develop 
the  subjective  MOEs.  One  survey  is  given  to  senior  Army  leadership  to  determine  the 
decision  weights  for  the  whole  hierarchy  and  the  second  survey  is  given  to  subject  matter 
experts  to  score  investment  alternatives.  The  AHP  is  used  to  make  the  comparative 
judgments.  The  AHP  breaks  down  a  problem  into  its  smaller  constituent  pans  and  then 
calls  for  only  simple  pairwise  comparison  judgments  to  develop  priorities  in  the 
hierarchy.  The  principle  of  comparative  judgments  calls  for  setting  up  matrices  to  carry 
out  pairwise  comparisons  of  the  relative  importance  of  elements  at  different  levels  of  the 
hierarchy. 

TOPSIS  is  an  algorithm  that  combines  both  the  simulated  combat  and  leader  subjective 
MOEs.  TOPSIS  characteristics  include:  a  rank-ordered  preference  list  of  all  candidate 
systems  from  a  constrained  set  of  initial  alternatives,  the  ability  to  compare  the  relative 
strength  of  preference  (on  an  ordinal  scale)  among  the  candidates,  a  set  of  quantifiable 
criteria  related  to  attributes  associated  with  each  alternative,  a  set  of  weights  which 
express  the  relative  measure  of  importance  the  decision  authority  attaches  to  each 
criteria,  a  degree  of  confidence  in  the  statistical  independence  and  reliability  of  the 
measures  associated  with  the  different  criteria,  and  a  requirement  for  sensitivity  measures 
associated  with  variance  in  input  values  or  decision  authority  tolerance  to  error.  TOPSIS 
output  is  used  in  the  formulation  of  the  objective  function  coefficient.  This  coefficient  is 
the  p>er-item  contribution  of  the  system  to  force  effectiveness  (vij).  Vij  is  maximized 
with  the  quantity  of  systems  procured  in  a  particular  year  (.xij)  in  the  optimization 
module.  FOMOA  subjects  the  maximized  function  to  budgetary,  force  structure,  and 
production  constraints. 
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2.4.  Outputs 

The  feasible  region  is  defined  by  the  constraints.  The  results  of  the  maximization 
produce  a  range  of  solutions  which  are  used  to  develop  an  acquisition  strategy. 

2.5.  Issues 

Issue  1;  What  is  the  context  of  the  problem  (purpose)  VAA  is  designed  to  solve? 

Response:  The  leadership  at  the  Department  of  the  Army  needs  analysis  to  support 
the  development  of  a  balanced  and  effective  Army  program  that  is  within  Department  of 
Defense  resource  guidance.  The  purpose  of  VAA  is  to  provide  the  Director  for  Program 
Analysis  and  Evaluation  (DPAE),  and  the  Deputy  Chief  of  Staff  for  Operations  and  Plans 
(DCSOPS)  with  an  analytical  tradeoff  methodology  and  capability  that  would  assist  the 
development  of  a  balanced  and  effective  Army  program  through  the  use  of  cost-benefit 
analysis[3]. 

Issue  2:  Does  VAA  provide  information  of  an  acceptable  type,  quality,  and  direction 
to  support  the  Army  Staff  in  the  program  evaluation  and  POM  development  effort? 

Background:  In  order  to  answer  this  question  we  need  to  explore  how  weapon 
systems  get  put  into  the  budget.  The  planning  phase  of  the  Planning,  Programming, 
Budgeting  and  Execution  System  (PPBES)  process  begins  with  Headquarters, 
Department  of  the  Army  (HQDA)  providing  guidance  for  the  development  of  a  Long- 
Range  Research,  Development  and  Acquisition  Plan  (LRRDAP).  The  LRRDAP 
establishes  the  strategy  for  force  modernization  (Research  Development  Test  and 
Evaluation  (RDTE)  and  procurement  appropriations)  for  the  POM  and  Extended 
Planning  Period.  The  LRRDAP  guidance  is  sent  to  all  Major  Army  Commands 
(MACOMs),  Army  Component  commanders,  and  Program  Executive  Offices  (PEOs)  for 
new  RDA  requirements  and  a  sensing  as  to  CINC/PEO  modernization  priorities.  It 
directs  the  field  proponents  (US  Army  Training  and  Doctrine  Command  (TRADOC) , 

US  Army  Materiel  Command  (AMC)  and  the  Information  Systems  Command  (ISC))  to 
develop  a  field  LRRDAP,  a  prioritization  of  all  research,  development  and  acquisition 
(RDA)  programs.  The  LRRDAP  is  prioritized  in  terms  of  Battlefield  Development  Plan 
(BDP)  deficiencies.  The  field  LRRDAP  is  submitted  to  the  DCSOPS  at  HQDA  for 
review  and  modification  [4],  VAA  is  used  in  this  review  and  modification  process  to 
analyze  competing  alternatives  that  fall  at  the  baseline.  The  field  LRRDAP  is  then 
approved  by  the  Chief  of  Staff  of  the  Army  (CSA)  at  a  Requirements  Review  prior  to  the 
POM  build  and  becomes  the  basis  for  RDA  requirements  in  the  POM. 

Response:  The  results  from  VAA  do  provide  information  of  an  acceptable  type,  and 
direction  to  support  the  Army  Staff  in  the  program  evaluation  and  POM  development 
process.  VAA  works  well  as  a  rapid  response  tool  and  is  vital  in  answering  "what  if..." 
questions  proposed  by  project  sponsors.  These  sponsors  and  the  VAA  analysts  work 
closely  together  in  determining  the  mix  of  systems  that  they  want  evaluated,  where  they 
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want  it  evaluated,  and  when  they  want  it  evaluated.  VAA  can  be  used  to  vary  the 
production  scheduling  of  different  systems,  provide  an  insight  to  decision  makers  in 
determining  what  systems  fall  out  when  changes  are  made  in  the  Total  Obligation 
Authority  (TOA),  and  provide  decision  makers  with  a  means  to  justify  or  lefiite 
preferences. 

The  quality  of  the  information  can  be  improved  by  eliminating  the  TOPSIS  algorithm 
from  the  methodology.  The  replacement  for  TOPSIS  could  be  in  the  form  of  an  additive 
or  multiplicative  measurable  value  function.  This  function  can  be  obtained  by  utilizing 
the  principles  of  multi-attribute  utility  theory  (MAUT). 

Issue  3:  Is  the  overall  architecture  of  the  methodology  well  defined?  What  are  the 
architectural  and  theoretical  foundations  used?  Do  these  foundations  adequately  match 
the  requirements? 

Response:  The  overall  VAA  architecture  is  well  defined.  A  hierarchical  assessment 
framework  is  used  to  measure  the  contribution  of  a  program  alternative.  The  framework 
is  designed  to  measure  the  effectiveness  of  the  alternative  both  explicitly  and  implicitly. 

It  combines  these  explicit  and  implicit  effectiveness  values  together  into  a  single  measure 
of  benefit .  The  combined  measure  is  evaluated  against  cost  using  a  multiobjective  linear 
program  to  produce  an  acquisition  strategy  based  on  cost-effectiveness. 

The  analytical  method,  AHP,  has  merit  in  its  contribution  to  analyzing  the  structure  of 
an  issue  and  in  quantifying  subjective  judgments.  AHP  provides  an  added  measure  of 
consistency  and  is  a  good  method  for  determining  weights.  There  have  been  some 
criticisms  within  the  analytic  community  with  regards  to  the  AHP  and  Saaty's  pairwise 
comparison  approach  when  alternatives  are  evaluated  using  this  procedure.  The 
criticisms  address  the  possibility  of  rank  reversal,  the  frame  of  reference,  and  the  scale  of 
measure. 

CORBAN  is  the  corps  level  combat  model  used  to  measure  the  combat  effectiveness  of 
systems.  CORBAN  focuses  on  overall  corps  effectiveness  and  capabilities  with  emphasis 
on  maneuver  actions,  activities,  and  weapon  systems[5].  TRADOC  Analysis  Center, 

Fort  Leavenworth  (TRAC-FLVN  )  is  the  proponent  for  CORBAN.  They  provided  the 
accreditation  package  for  CORBAN  to  the  US  Army  Combined  Arms  Command-Combat 
Developments,  Fort  Leavenworth  on  2  November,  1990.  The  package  contains  a 
complete  set  of  model  documentation,  an  overview  of  CORBAN's  validation  and 
verification,  an  information  paper,  and  a  configuration  control  policy.  The  model's 
strong  and  weak  points  are  included  in  the  information  paper.  The  paper  identified 
CORBAN's  strong  points  as  its  availability,  flexibility,  speed  and  low  staff  requirements. 
The  weak  points  were  identified  as  being  its  non-modular  structure  (which  made  it 
extremely  difficult  to  maintain  and  perform  structured  verification  and  validation),  and 
its  air  and  air  defense  portrayal[6]. 
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TOPSIS  is  used  to  combine  the  implicit  and  explicit  measures  of  value  into  a  single 
measure  of  benefit  for  use  in  the  optimization.  TOPSIS  normalizes  the  criteria  scores, 
weights  the  scores  based  upon  the  known  or  previously  determined  relative  importance 
of  each  of  the  criteria,  and  calculates  the  distance  from  an  Ideal  and  Negative  Ideal 
alternative.  These  extremes  are  used  as  standards  for  the  final  ranking  process.  The  best 
alternative  is  the  one  farthest  from  the  Negative  Ideal  and  closest  to  the  Ideal.  TOPSIS  is 
an  easy  to  use,  straight  forward  algorithm.  It  allows  attributes  of  different  units  to  be 
used  to  compare  alternatives  and  gives  a  clear  rank  ordering  of  the  alternatives  being 
considered.  The  TOPSIS  algorithm  has  several  weaknesses  associated  with  it  These 
weaknesses  are  the  possibility  of  rank  reversal,  arbitrary  ranking,  and  the  scaling  of  the 
output  coefficients.  The  rank  reversal  and  arbitrary  ranking  problems  should  be 
eliminated  if  the  modifications  to  the  TOPSIS  algorithm  were  made  [20].  The  major 
problem  with  the  algorithm  in  its  VAA  application  is  that  the  output  coefficients  consist 
of  rational  numbers  on  an  ordinal  scale.  This  means  the  coefficients  are  simply  rank 
ordered.  The  final  optimization  needs  coefficients  to  be  on  a  ratio  scale. 

The  flaws  within  the  structure  can  be  corrected  with  minimal  effort  and  should  produce 
more  precise  results. 

Issue  4:  Are  the  assumptions/limitations  specifically  stated  for  the  methodology  and 
are  they  reasonable? 

Response:  The  key  assumptions  for  the  overall  methodology  are  reasonable.  They 
are: 

1)  All  tradeoff  issues  will  only  be  those  "on  the  margin". 

2)  There  is  no  realignment  of  years  prior  to  the  current  POM  funding  levels. 

3)  VAA  would  only  be  used  to  measure  increments  and  decrements  starting 
from  a  base  represented  by  the  current  POM  (FY  92-97)  position. 

4)  TOA  guidance  may  or  may  not  be  developed  using  the  VAA  methodology. 

5)  The  methodology  must  allow  for  prioritizing  between  dissimilar  program 
alternatives. 
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3.  Verification 


This  section  illustrates  the  process  of  whether  the  models  and  simulations  used  in  the 
-methodology  accurately  represent  the  Concepts  Analysis  Agency's  (CAA)  conceptual 
description  and  specrfications[7]. 

3.1.  Issue  Clarification 

This  modules'  primary  purpose  is  the  identification  of  the  elements  of  the  tradeoff  issue 
and  any  issue-related  questions  or  problems.  The  elements  of  the  module  include  the 
receipt  of  an  issue,  its  description,  and  a  basic  understanding  of  where  the  issue  fits  in 
terms  of  national  goals  and  objectives. 

Figure  3.1  illustrates  where  the  issue  clarification  module  fits  in  the  overall  methodology. 


Figure  3.1.  Issue  Clarificalion 


Figure  3.2  illustrates  an 


example  of  an  araot/anti-aimoi  (A3)  system  trade-off. 


issus:  Given  limited  dollar  resources  which  mix  of  A3 
Systems  should  be  procured? 


/^ftamatlves 

Basecase 

Alt1 


o^^rf*rteristlcs 

Europe  Scenario  with  25  systems 

M1A2 
NLOS-AT 
MLRS  SADARM 


Alt  2 


M1A2 

AMS-H 

BRADLEY  BLK  IH 
MLRS  SADARM 


Figure  3.2.  RepresentaUve 


Armor/  AnU-Armor  (A3)  Systems 


3.1.1.  Input 

An  issue  is  set  up  based  upon  Mtecdon  ^esTconsiders  the  breakout 

year  or  scenario  combination  to  b  p  y  ■  alternative  systems.  It  also 

of  related  systems  wi*  them  ^  a„d  ,he  extent 

defines  the  scope  of  the  stuay  ,  tne  yp 
to  which  the  methodology  will  be  used. 

3.1.2.  Procedures 

The  procedures  are  primarily  ”8^“ 

method  developed  for  this  module[7]. 


3.1.3.  Outputs 

The  output  consists  of  murowing  »d 

guidance  may  be  important  to  the  decision  ^  the  year/scenario  to  be  played. 

Salysts  determine  the  list  of  ^  “  ^XStogetL  are  removed.  The 

srxtsir:;-  “ 

follow-on  explicit  effectiveness  module. 
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Figure  3.3  shows  an  example  of  the  form  in  which  the  data  may  be  presented[8]. 


Information  Matrix:  A3  Alternatives 


System 

AttrlbuteVy^ 

Tank  A 

Antitank 
Missile  A 

Submunition  B 

Procurement  Data 

Number  of  Lines 

2 

1 

5 

Number  of  Shifts 

1 

2 

1 

Hours/Shift 

6 

a 

6 

Systems/Month 

180 

54 

68 

Cost 

RDTE  ($M) 

170 

58 

43 

OMA($M) 

14.7 

4.5 

1.2 

MISM  Tree 

LIN 

B54690 
GA0 167 

C67900 

E66300 

A  MM  Priority 

6 

44 

23 

Figure  3.3.  Information  Matrix  of  A3  Systems 


3.1.4.  Issues 

Issue  1:  If  there  are  currently  45  different  systems  to  explore  can  all  possible 
combinations  of  these  systems  be  evaluated?  Does  the  possibility  exist  that  the  optimal 
mix  of  systems  may  be  discarded? 

Response:  The  requirement  to  evaluate  multiple  systems  in  various  combinations  is 
extremely  difficult  and  cannot  be  done  in  a  timely  manner  if  simulations  are  needed  for 
every  conceivable  gi  oiiping  of  weapon  systems.  CAA  determined  that  replacing  weapon 
systems  one  at  a  time  or  doing  every  possible  combination  of  systems  were  impractical 
methods  because  of  combinatorial  complexity.  Response  Surface  Methodology  (RSM) 
and  the  General  Linear  Model  (GLM)  were  chosen  to  find  the  best  combination  of 
systems  that  give  the  greatest  increase  in  the  measures  of  effectiveness  values.  RSM  is  a 
collection  of  mathematical  and  statistical  techniques  useful  for  analyzing  problems  in 
which  many  independent  variables  may  influence  a  dependent  variable  or  response.  The 
goal  of  RSM  is  to  optimize  that  response[91.  The  possibility  still  exists  that  the  optimal 
mix  of  systems  will  be  discarded,  but  it  is  remote. 
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Issue  2:  Are  the  right  issues  being  identified  and  clarified  and  who  decides  on  the 
issues  being  kept  or  discarded? 

Response:  Issues  are  identified  by  the  project  sponsors.  Clarification  of  issues  is 
performed  by  VAA  analysts  and  project  sponsors.  The  selection  of  issues  is  based  on 
the  importance  of  the  issue  to  the  decision  making  process  and  the  ability  to  do  a  credible 
job  modelling  the  systems  involved. 

Issue  3:  In  what  way  do  the  issues  drive  the  experimental  design  matrix? 

Response:  Issues  are  framed  by  the  selection  of  the  systems  to  be  considered,  the 
year/scenario  combinations  to  be  played,  and  can  involve  different  combinations  of 
systems.  The  combinations  of  systems  that  would  never  be  played  together  are  removed. 
The  systems  that  ultimately  drive  the  experimental  design  matrix  are  randomly 
generated[2]. 

Issue  4:  What  are  the  types  and  formats  of  the  required  input  data  for  the  issue 
clarification  module?  Is  the  input  data  appropriate  ap.d  available? 

Response:  Data  used  in  the  methodology  is  complex,  varied,  extensive  and  hard  to 
come  by  in  certain  instances.  The  format  for  the  data  can  either  be  narrative  or 
quantitative.  Data  on  existing  systems  and  on  future  systems  may  only  be  engineering 
data.  Policy  data  is  mostly  text  and  cost  data  is  predominantly  quantitative.  The  input 
data  used  in  VAA  is  appropriate  for  use  in  the  trade-off  analysis. 

Issue  5:  Are  the  assumptions  made  in  this  module  specifically  stated  and  are  they 
reasonable? 

Response:  There  were  no  assumptions  made  in  this  module[10],  but  there  should  be 
some  in  regards  to  the  discarding  of  alternatives  and  the  lack  of  available  data.  Since 
there  are  presently  up  to  45  different  systems  to  evaluated,  the  number  of  potential  runs 

would  be  2'^^  - 1  or  roughly  3.51  x  10^^  if  systems  are  to  be  evaluated  individually. 
RSM  alleviates  the  process  of  replacing  weapons  one  at  a  time  and  the  process  of  testing 
every  combination,  but  there  is  a  chance  that  the  optimal  solution  may  still  be 
overlooked.  Improving  the  range  of  alternatives  to  cover  variations  in  tactics,  materiel, 
and  organization  may  help  eliminate  the  possibility  of  the  optimal  solution  being  missed, 
but  an  assumption  should  be  made  to  that  effect.  The  lack  of  available  and  acquired  data 
is  a  problem  that  will  always  occur  no  matter  what  methodology  is  employed.  Data  will 
always  be  incomplete  or  even  contradictory  and  an  assumption  should  be  made  that  the 
lack  of  available  data  can  be  compensated  for  parametrically. 
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Recommendation. 


In  order  to  apply  the  principles  of  MAUT  to  the  methodology,  decisionmaker  preference 
(or  utility)  curves  should  be  developed  as  part  of  the  issue  clarification  module.  The 
purpose  for  these  curves  is  to  properly  scale  different  combat  and  non-combat  MOEs  in 
the  hierarchy  so  they  can  be  combined.  These  curves  also  have  the  added  benefit  of 
attempting  to  "capture"  the  risk  attitude  of  the  decisionmaker. 

The  most  important  issues  concerning  MOEs  throughout  the  methodology  lies  in  their 
comparison,  scale  of  measure,  weight  assigned,  and  combination.  Comparison  of  MOEs 
involves  analyzing  individual  MOE  values  for  each  alternative  and  determining  that  one 
value  is  a  higher  value  than  another.  The  scale  of  measure  involves  considering  the 
ranges  over  which  alternative  scores  are  based  and  aligning  MOEs  on  the  same  scale  so 
ratio  assessments  can  be  made.  The  weight  assigned  to  MOEs  must  vary  based  on  the 
importance  of  each  MOE  to  the  decision  making  process.  Combination  of  MOEs 
involves  integrating  the  explicit  and  implicit  MOEs  for  each  alternative  into  a  single 
measure  for  use  in  the  optimization. 

A  direct  comparison  of  individual  MOE  values  by  category  between  alternatives  is 
straightforward  since  MOEs  are  quantitative  measures.  One  can  just  look  at  a  single 
MOE  for  an  alternative  and  determine  if  that  value  is  better  than  the  corresponding  MOE 
value  for  another  alternative.  A  degree  of  difficulty  arises,  however,  when  alternatives 
must  be  compared  using  the  totality  of  all  MOEs.  MOEs  are  based  on  different  scales, 
have  different  ranges,  and  are  valued  differently  by  each  decisionmaker.  This  is  where 
the  additive  measurable  value  function  (AMVF)  can  be  used.  The  goal  of  using  the 
function  is  to  combine  MOEs  in  order  to  make  comparisons  among  alternatives.  It 
makes  use  of  decisionmaker  preference  curves  to  combine  the  MOEs.  A  separate 
preference  curve  can  be  developed  for  each  decisionmaker  for  each  MOE  by  asking  the 
decisionmaker  one  question(16].  These  preference  curves  can  be  developed  using  many 
different  methods.  The  direct  rating  method  is  the  one  illustrated  below. 

Let  the  following  scale  represent  the  MOE  effective  battalions  remaining  (EBR).  For 
simplicity,  the  scale  goes  from  0  to  ICKX),  where  0  is  no  battalions  remaining  (total 
defeat)  and  1000  is  all  battalions  remaining  (total  victory).  The  decision  maker  is  asked 
to  mark  numerically  what  he  would  consider  to  be  a  draw.  This  value  may  or  may  not 
fall  directly  in  the  middle  of  the  scale  as  illustrated  in  figure  3.7  below: 
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Figure  3.7.  Decisionmaker’s  Numeric  Scale 


The  decision  maker's  preference  function  can  now  be  plotted  for  the  MOE  EBR,  as 
shown  in  figure  3.8: 


Figure  3.8.  Decisionmaker's  Preference  Function  Curve 

The  u(x)  is  the  relative  score  of  the  alternative  on  the  MOE  EBR.  The  maximum  value 
for  u(x)  is  1,  the  draw  value  is  .5,  and  the  minimum  value  is  0.  Since  the  scale  is 
logarithmic  and  the  maximum,  draw,  and  minimum  values  for  u(x)  are  known  this 
enables  us  to  plot  and  capture  the  decision  maker’s  preference  curve.  All  MOEs  in  the 
hierarchy  can  be  scaled  and  compared  using  this  approach. 

The  weights  for  the  MOEs  are  obtained  using  the  AHP.  The  decision  maker  is  asked  to 
answer  a  series  of  pairwise  comparisons  regarding  the  importance  of  each  MOE.  He 
should  consider  the  ranges  of  the  alternatives  and  the  relative  importance  of  going  from 
the  least  preferred  to  the  most  preferred  values  for  each  MOE  The  combination  of  each 
alternative's  MOEs  can  then  be  performed  using  the  AMVF.  The  following  procedure 
can  be  used: 
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1)  MOFs  are  normalized  on  their  resoecrive  base  case.  This  procedure  is  performed  by 
tulcing  the  value  of  each  MOE  and  dividing  it  by  the  value  of  the  base  case  for  that  MOE. 
The  values  obtained  will  then  reflect  a  given  range.  The  upper  and  lower  endpoint 
values  for  all  the  normalized  MOEs  can  be  set  by  the  decision  maker  through  a  series  of 
questions.  The  upper  and  lower  endpoint  values  should  not  be  used  as  a  measure  of 
attributes  in  general,  but  should  be  a  reflection  of  going  from  the  worst  value  to  the  best 
value. 

2)  The,  decisionmaker^s  preference  function  curves  are  determined.  This  can  be  done  for 
the  MOEs  by  asking  the  decision  maker  one  question  per  MOE  as  previously  shown. 

The  normalized  MOE  values  for  all  alternatives  are  then  plotted  on  each  of  the  decision 
maker’s  utility  curves  and  all  the  u(x)  values  are  determined. 
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3)  .rp.  ...igned  to  the  MOEs.  The  weights  are  assigned  to  each  MOE  as 

determined  by  the  AHP.  These  weights  are^en  multiplied  by  the  corresponding  u(x) 
values  and  summed  by  the  additive  model  5.  wiUjj.  Where  wj  is  the  weight  of  cntenon 
i,  and  Ujj  is  the  relative  score  of  alternative  j  on  the  MOE  i. 

The  result  is  an  additive  multi-attribute  utility  (MAU)  function  which  is  unique  i-e.  any 
other  additive  model  using  the  same  attributes  that  ranks  alternatives  consistent  wiA  the 
decision  maker’s  preference  is  a  positive  linear  transformation  of  the  additive  MAU. 
benefit  of  this  additive  MAU  function  is  that  it  can  be  assessed  through  *e  use  of 
lotteries,  trade-off  questions,  difference  questions  or  ratio  judgments.  Al  emattves  can 
now  be  compared,  scaled,  weighted,  and  combined,  with  the  decision  maker  playing  a 

key  role. 
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3.2.  Costing 

This  module  identifies  all  appropriate  cx)Sts  associated  with  each  alternative  for  the  issue 
being  investigated.  Two  PC-based  models,  the  Cost  Quantity  Model  (CQM)  and  the  Life 
Cycle  Cost  Model  (LCCM)  are  used  to  couple  the  life  cycle  weapon  system(s)  cost 
elements  with  cost  elements  accounting  for  variations  in  procurement  quantities. 

Figure  3.9  illustrates  where  the  cost  clarification  module  fits  in  the  overall  methodology. 


3.2.1.  Input 

Life  cycle  cost  data  is  obtained  from  the  Baseline  Cost  Estimate  (BCE)  or  Army  Cost 
Positions  (ACP).  Army  Materiel  Command  (AMC)  provides  inflation  indices. 
Production  rates  are  obtained  from  the  Assistant  Secretary  of  the  Army  for  Research, 
Development,  and  Acquisition  or  the  Program  Managers. 

3.2.2.  Procedures 

The  CQM  estimates  the  experience  curve  for  a  given  weapons  system  through  the  use  of 
historical  production  cost,  quantity,  and  rate  data  and  computes  production  costs  given 
the  quantities  desired.  The  LCCM  redistributes  production,  fielding  and  sustainment 
costs  once  the  optimal  procurement  quantities  and  production  costs  are  calculated.  The 
LCCM  model  is  time-phased  by  appropriation,  and  the  life  cycle  costs  of  the  weapon 
system  are  computed  in  both  constant  and  current  dollars. 
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3.2.3.  Outputs 

weapoLysttinfa  breakdown  of  all  costs  by  approprianon,  and  total  procurement  fund 
costs  in  both  constant  and  current  dollars[l  11. 

3.2.4.  Issues 

Issue  1;  Can  the  derivation  of  the  most  sensitive  cost  elements  be  improved  without 
significant  computational  burden? 

Response;  The  most  sensitive  cost  elements  are  the  production  cost  elements.  The 

rOM  and  LCCM  are  useful  methods  for  estimating  the  impact  of  vanatio^  in 

on  ibe  life  cycle  c«  element  and 

nf  cost  elements  using  these  two  methods  needs  no  improvement.  However,  the  overall 
lX"^Ls'can  be  impreved  if  cos.  uncemindes.  and  the  renge  of  error  *ey 
are  likely  to  introduce,  are  incorporated  into  the  cost  estimate. 

Cost  uncertainty  can  be  classified  into  two  main  categories:  a  requirements  uncertainty, 
and  a  cost  estimating  uncertainty.  Requirements  uncertainty  can  be  attnbuted  to  system 
SnfiSon  changi.  For  example,  the  original  hardware  design  may  fail  to  meet  the 
desirSi  performance  characteristics  and  the  hardware 

This  uncertainty  can  be  dealt  with  using  parametnc  costing  methods.  Cost  estimat  g 
variations  in  the  cost  estimates  themselves.  For  example,  the  errom 
in  the  avaUable  cost  data  used  to  develop  cost  estimating  relationships.  This  uncertainty 
is  inherent  in  the  V.'^A  methodology  because  projections  of  future  weapon  system  cos  s 
are  based  on  historical  data.  The  impact  of  uncertainty  may  eliminate  certain 
from  consideration  and  change  the  procurement  scheduling  of  a  v^ety  of  systems, 
magnitude  of  the  uncertainty  for  each  system  has  to  be  determined. 

The  anpUcation  of  chance-constrained  programmings  could  be  used  to  admit  ^dom  data 
^SnsSrpermit  constraint  violations  up  to  sp^^  This  methodto^s 

deterministic  linear  constraints  which  can  be  used  to  define  a  feasible  regon.  Constraints 
can  be  determined  by  sampling  a  distribution-free  chance  constraint  set  from  within  a 
tol/rance  .^gfon  (Allen.  Braswell,  and  Rao,  1974).  Range  ereor 
estimates  can  be  made  using  genemliaed  statistical  estimating  techniques  and  confidence 

intervals. 


3Maior  Joseph  A.  Waldron,  USMA,  Department  of  Systems  Engineering. 
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Issue  2:  Who  obtains  the  information  from  the  BCE  and  ACP?  How  accurate  are  the 
BCE  and  ACP  for  life  cycle  cost  data? 

Response:  CAA,  in  coordination  with  the  US  Army  Cost  and  Economic  Analysis 
Center  (CEAC),  collect  all  cost  data.  The  BCE  and  ACP  are  the  data  available  and  come 
from  the  program  manager.  CAA  updates  the  data  as  revisions  are  needed  and  new  data 
becomes  available[2]. 

Issue  3:  Are  the  assumptions  in  this  module  specifically  stated  and  are  they 
reasonable? 

Response:  The  assumptions  stated  are  reasonable  and  are[121: 

1)  Economies  of  scale  are  reflected  by  existing  or  derived  composite  learning 

curves. 

2)  Analogous  system  may  be  used  for  future  systems  when  future  system  cost 
data  are  not  available. 

3)  BCE  and  ACP  data  are  used  for  Base  Case  unit  production  cost  and  quantities, 
when  available. 

4)  Cost  quantity  relationships  exist  as  defined. 

5)  Component  driven  models  ^e  impractical  in  a  quick  reaction  environment. 

6)  Regression  analysis  of  historical  or  projected  annual  weapon  system 
production  costs  can  be  used  to  calculate  system  first  unit  costs. 

7)  Experience  and  production  rate  curves  that  generate  average  annual  production 
cost  can  be  determined  within  an  acceptable  margin  of  error. 

8)  No  contractual  penalties  are  incurred  if  production  quantities  are  changed. 

9)  O&S  costs  are  based  on  cumulative  fielding  of  systems  and  begin  when  the 
first  system  produced  is  fielded. 
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3.3.  Explicit  Effectiveness 

Three  submodules  comprise  this  module;  combat,  soldier  quality  of  life,  and  other  Army 
objectives.  The  combat  submodule  is  the  only  submodule  that  is  analyzed  due  to  the 
limited  scope  of  the  VAA  Phase  I  study.  The  context  of  the  combat  submodule  is  based 
upon  the  scenarios  modeled,  force  structure,  and  doctrine  employed.  The  overall  purpose 
of  the  combat  submodule  is  to  measure  combat  effectiveness. 

Figure  3.10  illustrates  where  the  explicit  effectiveness  module  fits  in  the  overall 
methodology. 


issue  Clarification 
Module 


implicit 

Effectiveness 

Module 


Cost 

Module 

1 

I  Effectiveness 
Integration 
Module 


I  Optimization 
Module 


Figure  3.10.  Explicit  Effectiveness 


3.3.1.  Input 

CORE  AN  is  the  model  used  for  determining  combat  effectiveness.  The  input  to 
CORBAN  consists  of:  scenario,  terrain,  representative  Red  Forces  and  Blue  Forces, 
missions,  and  orders.  This  includes  asset  characteristics  such  as  target  types,  fire  rate, 
probability  of  acquisition,  ammo  type,  fuel  use,  crew  levels,  vulnerabilities,  search 
patterns,  and  sensor  classes. 

3.3.2.  Procedures 

RSM  was  chosen  to  take  the  place  of  the  case  by  case  introduction  of  alternatives.  The 
RSM  technique  takes  specific  combinations  of  the  total  combinations  possible.  The 
combination  results  are  averaged  to  find  an  estimate  for  a  system's  contribution  to  the 
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measure  of  effectiveness[13].  RSM  uses  a  design  matrix  that  varies  the  inputs  so  that  a 
linear  model  can  be  built  to  forecast  the  effects  of  the  systems  with  respect  to  the  outputs. 
The  average  improvement  given  the  new  weapon  system  takes  the  form  of  a  set  of 
coefficients. 

3.3.3.  Outputs 

The  MOEs  that  CORBAN  analysts  most  commonly  use  are:  correlation  of  forces  and 
means  (COFM),  force  ratios  (FR),  loss  exchange  ratios  (LER),  fractional  exchange  ratios 
(PER),  systems  effectiveness  ratios  (SER),  killer-victim  table,  effective  battalions 
remaining,  movement  of  force  center  of  mass,  mission  accomplishment,  front  line  trace, 
and  plots  of  unit  locations  [14]. 

3.3.4.  Issues 

Issue  1:  What  are  some  of  the  problems  associated  with  the  MOEs?  Are  there 
MOEs  which  are  able  to  provide  better  operational  insights?  Do  the  MOEs  reflect  the 
issues  or  the  systems  under  consideration? 

Response:  MOEs  are  criteria  expressing  the  extent  to  which  a  combat  system 
performs  a  task  assigned  to  that  system  under  a  specified  set  of  conditions.  The  MOEs 
should  depend  on  the  possibility  of  quantifying  or  modeling  the  weapon  system 
objective[15].  The  MOEs  that  analysts  most  commonly  use  are  too  dependent  and  very 
similar  in  nature.  For  example,  the  LER  is  the  measure  of  the  combat  value  of  Red 
systems  lost  to  the  combat  value  of  Blue  systems  lost  and  the  FR  is  the  measure  of  the 
overall  Red  forces  remaining  with  respect  to  the  overall  Blue  forces  remaining.  The  FER 
is  the  ratio  of  proportion  of  Red  systems  value  lost  to  the  proponion  of  Blue  systems 
value  lost.  SER  is  a  ratio  of  the  number  of  Red  systems  killed  by  a  blue  system  to  the 
number  of  Blue  systems  killed.  Killer-victim  tables  show  losses  and  killers  by  asset  and 
asset  category  but  are  not  considered  MOEs. 

Recommendation:  MOEs  used  should  be  reflective  of  the  issues  or  systems  under 
consideration.  MOEs  that  may  be  incorporated  into  the  combat  model  include 
percentage  of  blue  force  casualties  and  combat  system  utilization^  .  Percentage  of  blue 
force  casualties  is  the  ratio  of  blue  force  casualties/  initial  blue  force  strength.  Combat 
system  utilization  is  the  percentage  of  kills  by  a  system/  the  percent  of  those  systems  in 
the  force.  The  MOE,  LER,  is  redundant  and  can  be  eliminated. 

Issue  2:  Are  there  any  credibility  issues  concerning  CORBAN? 

Response:  TRAC-FLVN  stated  in  the  accreditation  package  for  CORBAN  that  they 
were  not  aware  of  any  functional  area  concerns  over  model  credibility  by  any  of  the 
schools  or  centers[6]. 
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Issue  3;  What  are  some  of  the  problems  associated  with  RSM? 

Response:  RSM  gives  combat  estimates  with  respect  to  different  scenarios  and  years. 
The  coefficients  are  a  reflection  of  those  systems  in  battle.  The  main  problems 
associated  with  RSM  involve  understanding  and  explaining  some  of  the  results  that  are 
obtained.  CAA  noted  that  in  every  case  in  which  these  strange  or  unusual  results  have 
occurred  the  cause  has  been  from  data  input  errors. 

RSM  does  provide  for  the  amount  of  replication  needed  to  achieve  sufficient  precision  as 
long  as  the  location  of  the  region  of  interest  is  properly  identified  and  the  appropriate 
scalings  and  transformations  for  the  input  and  output  variables  are  made[17]. 

Issue  4:  Are  the  assumptions  in  this  module  specifically  stated  and  are  they 
reasonable? 

Response:  The  assumptions  stated  are  reasonable  and  are  that[12]: 

1)  Two  years  after  the  end  of  the  POM  is  when  the  equipment  will  arrive  in  units. 

2)  Current  M+10  force  will  be  used  in  both  1999  and  2004. 

3)  COFM  is  the  measure  of  the  Red  Force  commanders'  perception  of  success. 

4)  BFS  is  the  Blue  Force  commanders'  ability  to  prosecute  his  plan. 

5)  Quantities  modeled  are  similar  to  quantities  bought  so  new  effectiveness 
simulations  do  not  need  to  be  run. 

There  are  several  assumptions  that  must  be  made  to  obtain  the  AMVF  for  use  in  this 
module[16].  The  first  assumption  regards  the  strength  of  preference  the  decision  maker 
has  for  an  alternative.  This  is  where  the  decision  maker  must  articulate  how  much  more 
he  prefers  one  alternative  over  another.  The  second  assumption  regards  the  ordering  of 
expressions  of  preference  of  the  decision  maker.  For  example,  if  the  decision  maker's 
preference  for  ^temative  A  over  alternative  B  is  greater  than  his  preference  for 
alternative  C  over  alternative  D,  then  his  preference  for  having  to  accept  alternative  D 
over  alternative  C  should  be  greater  than  his  preference  for  having  to  accept  alternative  B 
over  alternative  A.  The  third  assumption  is  that  the  decision  maker’s  preferences  satisfy 
the  properties  of  connectedness.  This  means  that  if  the  decision  maker  prefers  alternative 
a  to  alternative  b,  then  he  can  not  also  prefer  alternative  b  to  alternative  a.  The  fourth 
assumption  is  that  the  decision  maker's  preferences  satisfy  the  property  of  transitivity. 
Transitivity  implies  that  if  the  decision  maker  prefers  alternative  a  to  alternative  b  and  he 
prefers  alternative  b  to  alternative  c,  then  he  prefers  alternative  a  to  alternative  c.  The 
fifth  assumption  regards  preference  independence  the  decision  maker  has  for  an 
alternative.  The  best  way  to  explain  preference  independence  is  with  an  example.  Let  us 
assume  we  have  two  alternatives  and  we  wish  to  show  they  are  preference  independent. 
These  alternatives  are  alternative  a  and  alternative  b  respectively.  Let  us  also  assume  that 
only  three  MOEs  result  from  running  CORBAN.  These  MOEs  are  FER,  EBR,  and 
COFM.  We  can  write  these  alternatives  in  terms  of  their  MOEs  as  vectors,  a  =  (ai,a2,a3) 
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and  b  =  (bi,b2,b3)  (where  aj  was  the  FER  for  alternative  a  etc...).  If  the  decision  maker 
prefers  (ai,a2,a3  )  to  (bj,b2,a3),  he  indicates  that  he  prefers  the  FER  and  EBR  values  for 

alternative  a  over  those  for  alternative  b.  He  does  not  wish  to  consider  the  COFM  values 
for  alternative  b  and  only  likes  those  COFM  values  from  alternative  a.  He  preferences 
are  independent  if  and  only  if  he  now  prefers  (ai,a2,b3  )  to  (bi,b2,b3 )  i.e,  he  would  still 
prefer  FER  and  EBR  from  alternative  a  over  alternative  b  if  the  COFM  values  for 
alternative  a  were  not  considered. 

If  it  is  determined  that  the  decisionmaker’s  utility  function  is  not  logarithmic  in  nature, 
then  we  must  assume  we  can  determine  the  function  through  the  use  of  lotteries.  Once 
all  of  these  assumptions  are  met,  the  AMVF  from  MAUT  can  be  used  to  scale,  compare, 
and  combine  the  MOEs  generated  from  CORBAN. 
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3.4.  Implicit  Effectiveness 

This  module  has  two  objectives:  weighting  all  the  levels  of  the  assessment  hierarchy  (see 
appendix  b)  and  individual  criteria,  and  scoring  each  alternative  using  Secondary  Impact 
Analysis  Modifiers  (SIAM)  factors.  These  SIAM  factors  are  subjective  decision  criteria 
(see  appendix  c).  The  module  is  designed  to  quantify  the  subjective  elements  of  the 
decision  process  so  they  can  be  used  to  conduct  tradeoff  analyses 

Figure  3.1 1  illustrates  where  the  implicit  effectiveness  module  fits  in  the  overall 
methodology. 


3.4.1.  Input 

An  Army  leadership  survey  is  used  to  weight  all  the  levels  of  the  assessment  hierarchy 
and  individual  criteria.  A  subject  matter  expert  survey  is  used  to  score  investment 
alternatives  using  the  SIAM  factors. 

3.4.2.  Procedures 

The  establishment  of  relative  strength  or  priority  (weights)  is  accomplished  using  Saaty's 
pairwise  comparison  technique.  A  ratio  judgment  is  made  using  these  scores.  The 
weights  are  computed  using  the  geometric  mean  method  and  normalized  so  they  sum  to 
one.  Subjective  judgments  are  made  by  subject  matter  experts  on  the  relative  value  of 
weapons  systems  with  respect  to  SIAM  factors.  These  judgments  are  also  made  using 
Saaty’s  pairwise  comparison  technique.  The  number  of  judgments  is  based  upon  the 

number  of  alternatives  (for  n  alternatives,  an  n  x  n  matrix  of  (n^'  -  n)/2  judgments  is 
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developed  i.e,  if  n=7  there  are  21  judgments).  Experts  rate  each  of  the  weapon  systems 
using  verbal  comparisons.  This  verbal  interpretation  is  translated  to  a  numeric 
interpretation  on  a  scale  from  1  to  9.  A  ratio  judgment  is  then  made  using  the  scores. 

3.4.3.  Outputs 

The  output  consists  of  a  rank  ordering  of  alternatives  to  be  used  in  the  effectiveness 
integration  module.  The  TOPSIS  algorithm,  in  the  effectiveness  integration  module, 
combines  these  rankings  with  the  explicit  effectiveness  rankings  and  comes  up  with  a 
single  measure  of  effectiveness. 

3.4.4.  Issues 

Issue  1:  There  have  been  some  criticisms  within  the  analytic  community  with  regards 
to  the  AHP  and  Saaty's  pairwise  comparison  approach.  The  general  criticisms  address 
the  possibility  of  rank  reversal,  the  frame  of  reference  and  scale  of  measure,  and  the 
comparison  of  AHP  with  traditional  utility  theory  (UT).  How  valid  are  these  criticisms 
and  are  they  applicable  to  the  VAA  methodology? 

Background:  The  possibility  exists  that  rank  reversal  may  occur  when  using  a 
technique  based  on  hierarchic  composition.  The  following  example  by  Dyer  and 
Wendell  illustrates  the  ranking  reversal  problem[18].  It  should  be  noted  that  this 
example  of  rank  reversal  is  occurring  under  specific  conditions  i.e,  equal  weights  are 
assigned  to  all  criteria. 


Four  alternatives  alt  1,  alt  2,  alt  3,  and  alt  4  are  compared  against  four  criteria  Ci,C2,C3^ 
and  C4  by  a  single  decisionmaker  on  an  arbitrary  scale  from  1  to  9  (see  table  3.4.1) 


Cl 

C2 

C3 

C4 

alt  1 

1 

9 

1 

3 

alt  2 

9 

1 

9 

1 

alt  3 

8 

1 

i 

4 

5 

alt  4 

4 

1 

8 

5 

Table  3.4.1.  Rank  Reversal  Problem 
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For  simplicity,  we  assuming  that  the  four  criteria  are  judged  to  be  equally  important 
(same  weight),  the  rankings  for  the  first  three  alternatives  are  shown  in  table  3.4.2  below: 


Cl 

C2 

C3 

C4 

Score 

Rank 

altl 

1/18 

9/11 

1/14 

3/9 

.320 

3 

alt  2 

9/18 

1/11 

9/14 

1/9 

.336 

2 

alt  3 

8/18 

1/11 

4/14  ^ 

5/9 

.334 

I 

Table  3.4.2.  AHP  Rankings 

The  column  normalized  matrix  values  are  obtained  by  taking  the  original  scored  entries 
and  dividing  them  by  the  sum  of  the  entries  in  their  particular  column. 

Adding  the  fourth  alternative  creates  the  effect  shown  in  table  3.4.3: 


Cl 

C2 

C3 

C4 

Score 

Rank 

altl 

1/22 

9/12 

1/22 

3/14 

.264 

1 

alt  2 

9/22 

1/12 

9/22 

1/14 

.243 

4 

alt  3 

8/22 

1/12 

4/22 

5/14 

.246 

2 

alt  4 

4/22 

1/12 

8/22 

5/14 

.246 

2 

Table  3.4.3.  AHP  Addition  of  Fourth  Alternative 
where  alternatives  1  and  3  have  reversed  rankings. 

An  inspection  of  the  results  provides  some  insights  into  the  problems  that  are  inherent  in 
hierarchic  composition.  In  order  to  obtain  the  scores  shown  above,  the  numbers  in  the 
columns  are  each  multiplied  by  .25  (reflecting  the  assumption  that  the  criteria  are  equally 
important)  and  summed  across  the  rows.  The  reason  rank  reversal  occurs  is  because 
when  only  three  alternatives  alt  1,  alt  2,  and  alt  3  are  considered,  8/18  of  the  weight  on 
criteria  C  i  is  allocated  to  alt  3  by  the  first  term  in  the  calculation  of  the  score  for  alt  3, 
and  9/1 1  of  the  weight  on  criteria  C2  is  allocated  to  alt  1  by  the  second  term  in  its  sccro 
calculation.  Alt  3  has  higher  scores  than  alt  1  on  criteria  C3  and  C4  which  are  enough  to 
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give  it  a  higher  ranking  than  alt  1.  When  the  new  fourth  alternative  is  introduced  it  does 
not  do  well  on  criteria  C^,  C3,  and  C4,  which  is  where  alt  3  gets  most  of  its  allocated 

score  in  the  case  of  the  three  alternatives.  Therefore  it  dilutes  the  allocation  of  the  scores 
of  these  criteria.  Since  alt  I  performed  poorly  across  these  criteria,  it  did  not  suffer 
significantly  because  it  had  such  a  small  portion  of  this  weight  initially.  However  alt  4 
has  poor  perfonnance  on  criteria  C2  where  alt  1  excels,  so  the  fraction  of  the  weight  of 

C2  allocated  to  alt  1  falls  from  9/1 1  to  9/12.  As  a  result,  alt  3  is  hurt  more  by  the 
introduction  of  alt  4  than  alt  1,  and  a  rank  reversal  occurs. 

Response:  Since  the  AHP  is  used  and  it  is  based  on  a  hierarchical  structure,  the 
possibility  exists  that  rank  reversals  can  occur.  There  is  a  great  deal  of  concern  in  the 
analytic  community  regarding  the  frame  of  reference  and  scale  of  measure  used  in 
making  pairwise  comparisons.  Decisionmakers  arc  asked  to  reply  to  questions  on  a 
verbal  scale  and  the  verbal  response  is  then  translated  into  a  number  from  1  to  9.  This 
number  may  not  be  reflective  of  the  true  feelings  of  the  decision  maker.  For  example, 
the  scale  value  of  3  implies  'weak'  importance  of  one  objective  over  another  in  a  verbal 
description,  but  the  3  rating  means  one  objective  is  3  times  more  important  than  the 
other.  This  assignment  of  'weak'  importance  to  a  weight  of  3  is  not  justifiable.  These  1 
to  9  ratings  may  also  vary  from  decisionmaker  to  decisionmaker.  The  assigned  value  of 
6  from  one  decisionmaker  may  only  translate  to  an  assigned  value  of  4  by  another 
decisionmaker.  The  scale  used  lacks  a  ratio  interpretation  in  the  sense  that  only  numbers 
from  1  to  9  are  used  and  an  assigned  value  of  zero  is  impossible  to  make. 

There  may  be  a  problem  if  the  normalization  procedure  used  for  determining  the  weights 
in  the  hierarchy  is  used  for  evaluating  the  relative  sizes  of  alternatives.  Use  of  this 
procedure  may  eliminate  the  ratio  interpretation  of  the  alternatives  which  was  obtained 
from  the  surveys.  The  following  example  illustrates  how  this  normalization  is  performed 
and  where  the  problem  lies: 

A  single  decisionmaker  compares  three  alternatives  against  the  same  three  alternatives 
using  Saaty's  pairwise  comparison  technique  and  the  following  matrix  is  determined: 


A 

B 

C 

A 

1 

5 

1 

B 

1/5 

1 

1/5 

C 

1 

5 

1 

Table  3.4.4.  Saaty's  Pairwise  Comparison  Technique 
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This  matrix  is  called  a  reciprocal  matrix  because  it  has  positive  entries  ev^where  and 
satisfies  the  reciprocal  property  ajj  =  1/ajj.  The  scale  used  in  the  formulation  of  the 
matrix  goes  from  1  to  9.  Reciprocals  are  obtained  when  comparing  alternative  versus 

alternative.  (Using  the  above  matrix,  a  strong  preference  for  alternative  C  over 

alternative  B  results  in  an  assignment  of  5  in  the  row  C  column  B  position. 
Corresponding  to  that  entry,  the  reciprocal  value  of  1/5  is  assigned  to  the  row  B  column 

C  entry.) 

Using  these  assigned  values,  the  geometric  mean  vector,  V ,  is  then  computed  to  be: 

v=. 


((1X5X1))”* 

1.70»7 

1/s 

({1/5X1X1/5)) 

.34200 

((1X5X1))"* 

t.70997 

—  — 

—  — 

The  normalized  geometric  mean  vector,  V^,  is  then  formed  with  the  V  values: 


.4545 

.0909 

.4545 


Notice  that  the  sum  of  the  elements  of  Vjj  sum  to  1.  These  values  should  only  sum  to  I 
if  they  are  used  for  weighting  the  hierarchy  and  not  for  evaluating  alternatives.  If  they 
are  used  for  evaluating  alternatives,  the  scale  will  lose  some  of  its  ratio  interpretation. 

The  values  of  for  alternatives  could  have  been  values  such  as: 


_  — 

—  — 

2250 

.9000 

.0450 

or 

.1800 

.2250 

^000 

These  values  are  in  the  same  ratio  as  the  previous  Vr  and  there  is  no  reason  why  they 
cannot  be  used. 
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If  this  is  the  method  used  to  evaluate  alternatives,  a  suggestion  to  correct  this  problem 
would  be  to  apply  the  principles  of  MAUT  to  rescale  the  normalized  geometric  mean 
vector.  This  application  of  MAUT  assigns  the  maximum  value  a  1  and  the  minimum 
value  a  0  which  maintains  the  ratios  that  have  been  determined  using  the  AHP.  The 
following  rescaling  steps  should  be  then  be  taken: 

1)  Take  the  normalized  scores  from  AHP  [  .4545,  .0909,  .4545]. 

2)  Subtract  the  lowest  score  from  all  of  the  scores  [.4545  -  .0909,.0909  -  .0909 ,.4545  - 
.0909]  =  [.3636,0,  .3636]. 

3)  Divide  this  result  by  the  largest  number  [.3636/.3636,0/.3636, .36367.3636]  =  [1,0,1] 

This  rescaling  is  a  translation  of  the  normalized  scores  and  gives  the  best  alternative  a  1 
and  the  worst  alternative  a  0.  The  final  product  is  an  AHP  ratio  assessment  which  results 
in  an  additive  MAU  function.  The  values  obtained  using  this  MAU  function  can  be 
directly  combined  with  the  scaled  explicit  effectiveness  MOEs  in  the  effectiveness 
integration  module. 

The  comparison  of  utility  theory  with  the  AHP  is  not  a  valid  comparison  to  make.  AHP 
has  no  risk  involved  in  its  assessment  techniques.  The  questions  involved  in  the  risky 
case  should  involve  risk  and  should  try  to  capture  the  decision  makers  attitude  toward 
risk,  incorporating  that  attitude  in  the  assessment  technique. 

Issue  2:  Are  the  assumptions  in  this  module  specifically  stated  and  are  they 
reasonable? 

Response:  The  assumptions  made  for  this  module  are[12]: 

1)  The  objective  function  for  the  SIAM  factors  are  representative  of  the  decision 
function. 

2)  The  objective  function  for  each  of  the  SIAM  factors  is  correct. 

The  assumptions  made  in  the  explicit  effectiveness  module  need  to  carry  over  to  this 
module  if  MAUT  is  applied. 
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3.5.  Effectiveness  Integration 

This  mcxiule  combines  the  implicit  and  explicit  effectiveness  values  with  the  decision 
makers'  weights  and  integrates  these  multiple  measures  of  effectiveness  into  a  single 
measure  of  value.  This  value  is  then  used  to  conduct  a  cost  effectiveness  evaluation. 

Figure  3. 12  illustrates  where  the  effectiveness  integration  module  fits  in  the  overall 
methodology. 


Figure  3.12.  Effectiveness  Integration 


3.5.1.  Input 

The  input  consists  of:  soldier  quality  of  life  scores,  CORBAN  effectiveness  scores,  other 
Army  objectives  scores,  and  the  implicit  rank  order  of  alternatives.  The  input  is  in  the 
form  of  an  ASCII  text  file  which  is  commonly  prepared  by  using  a  microcomputer 
spreadsheet  program. 

3.5.2.  Procedures 

The  TOPSIS  value  measures  the  distance  of  an  alternative  from  an  ideal  solution.  This  is 
the  single  measure  of  value  and  becomes  the  effectiveness  coefficient  in  the  objective 
function.  TOPSIS  characteristics  include:  a  rank-ordered  preference  list  of  all  candidate 
systems  from  a  constrained  set  of  initial  alternatives,  the  ability  to  compare  the  relative 
strength  of  preference  (on  an  interval  scale)  among  the  candidates,  a  set  of  quantifiable 
criteria  related  to  attributes  associated  with  each  alternative,  a  set  of  weights  that  express 
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the  relative  measure  of  importance  the  decision  authority  attaches  to  each  criteria,  a 
degree  of  confidence  in  the  statistical  independence  and  reliability  of  the  measures 
associated  with  the  different  criteria,  and  a  requirement  for  sensitivity  measures 
associated  with  variance  in  input  values  or  decision  authority  tolerance  to  error. 

3.5.3.  Outputs 

The  output  consists  of  the  TOPSIS  coefficient  and  the  relative  ranking  among  the  listed 
alternatives.  The  TOPSIS  coefficient  is  used  in  the  optimization  module  as  a  means  of 
calculating  the  cost  effectiveness  evaluation.  This  value  can  also  be  used  to  rank  order 
alternatives. 

3.5.4.  Issues 

Issue  1:  Does  the  TOPSIS  methodology,  as  currently  implemented,  reflect  a  suitable 
and  intrinsically  meaningful  process  for  the  generation  of  value  coefficients  for  use  in  the 
optimization? 

Response:  The  use  of  Ideals  and  Negative  Ideals  has  some  advantages.  The  entire 
space  between  the  two  ideals  defines  the  feasible  region.  This  enhances  sensitivity  and 
aids  in  differentiating  between  alternatives.  TOPSIS  is  easy  to  use,  simplistic  in  design, 
and  gives  a  clear  ranking  of  alternatives.  The  output  from  the  TOPSIS  algorithm, 
however,  is  not  appropriate  for  use  in  the  objective  function.  This  relative  ranking 
consists  of  coefficients  that  are  based  upon  an  ordinal  scale. 

Issue  2:  Is  the  relative  ranking  from  the  TOPSIS  algorithm  an  appropriate  coefficient 
for  the  objective  function? 

Response:  Several  weaknesses  of  the  TOPSIS  algorithm  exist  that  make  it 
inappropriate  for  use  in  the  objective  function.  These  weaknesses  are  rank  reversal, 
arbitrary  ranking,  and  inappropriate  scaling.  The  following  example  shows  the  problem 
with  rank  reversal  and  arbitrary  ranking: 
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Using  a  base  case  with  three  alternatives  (see  table  3.5.1),  random  number  TOPSIS  mput 
may  be  [20]: 


Combat 

Effectiveness 

Soldier  Quality 
of  Life 

Secondary 
Impact  Analysis 
Modifiers 

(-) 

<*> 

(+) 

base  case 

3 

250 

100 

altl  ^ 

4 

350 

95 

all  2 

3 

350 

87 

alt  3 

1 

339 

91 

Table  3.5.1.  TOPSIS  Rank  Reversal. 

mse  values  are  then  normalized  and  weighted.  The  TOPSIS  scores  are  shown  in  table 
3.5.2  below: 


Results 

D* 

D- 

TOPSIS  Score 

base  case 

.067612 

.086587 

361525 

altl 

.127548 

.012852 

.091535 

all  2 

.104527 

.033806 

.244383 

all  3 

D69978 

.101974 

^93037  ** 
ranked  number 

1 

Table  33.2  Normalized  TOPSIS  Values 


The  D*  and  D*  indicate  the  distance  from  the  Ideal  and  Negative  Ideal  respectively. 
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The  addition  of  a  fourth  alternative,  alt  4,  leads  to  the  following  (see  table  333): 


Combat 

Effectiveness 

Soldier  Quality 
of  Life 

Secondary 
Impact  Analysis 
Modifiers 

(-) 

(-) 

(+) 

base  case 

3 

250 

100 

altl 

4 

350 

95 

alt  2 

3 

350 

87 

alt  3 

1 

339 

91 

alt  4 

5 

275 

95 

Table  3^.3  TOPSIS  Addition  of  Fourth  Alternative 


The  results  from  TOPSIS  are  shown  in  table  3.5.4  below: 


Results 

D* 

D* 

TOPSIS  Score 

base  case 

.051640 

.089626 

.634449  •* 
ranked  number 

1 

altl 

.105218 

J028247 

.211641 

alt2 

.089626 

D51640 

J6555I 

alt  3 

.064359 

.103731 

.617116 

alt  4 

.105032 

.054358 

.341037 

Table  33.4  TOPSIS  Results 
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The  addition  of  a  less  optimum  alternative,  alt  4,  changed  the  position  of  the  top  two 
alternatives.  This  is  due  to  alt  4  changing  one  of  the  negative  ideal  boundaries. 
Therefore  the  ranking  is  not  preserved. 

This  problem  can  be  eliminated  by  establishing  a  fixed  Ideal  and  Negative  Ideal  [20], 

Issue  3:  Is  there  another  method  of  combining  effectiveness  values  into  a  single 
measure  that  may  be  used  in  lieu  of  the  TOPSIS  algorithm? 

Response:  The  AMVF  (shown  previously)  allows  for  the  combination  of  the  u(x) 
effectiveness  values  from  the  explicit  and  implicit  effectiveness  modules.  These  u(x) 
values  are  multiplied  by  the  weights  obtained  using  the  AHP  and  then  summed  for  each 
alternative.  The  alternatives  are  then  rank  ordered.  This  can  all  be  performed  without 
using  the  TOPSIS  algorithm. 

Issue  4:  Are  the  assumptions  in  this  module  specifically  stated  and  are  they 
reasonable? 

Response:  There  were  no  assumptions  made  in  this  module  but  the  same 
assumptions  that  were  made  in  the  explicit  and  implicit  effectiveness  module  apply  to 
this  module. 


33 


3.6.  Optimization 


This  module  is  used  to  conduct  the  cost  effectiveness  evaluation.  It  allows  the  user  to 
simultaneously  trade  off  alternatives  based  on  their  cost  effectiveness  and  schedule  an 
alternative’s  acquisition. 

Figure  3.13  illustrates  where  the  optimization  module  fits  in  the  overall  methodology. 


The  rank  ordered  data  coefficients  from  TOPSIS  are  used  to  form  the  objective  function 
coefficient.  This  coefficient  is  maximized  with  the  quantity  of  the  system  procured  in  a 
given  year.  This  maximization  is  subjected  to  budgetary,  force  structure,  and  production 
constraints  using  FOMOA.  The  result  is  a  feasible  acquisition  strategy. 

3.6.2.  Procedures 

Three  sets  of  constraints  are  used  to  maximize  effectiveness.  The  first  set  insures 
procurement  expenditures  will  be  no  greater  than  the  total  obligation  authority.  The 
second  set  minimizes  (or  maximizes)  the  number  of  each  system  needed  based  on  force 
structure  requirements.  The  third  set  addresses  the  total  number  of  systems  of  each  type 
that  can  be  economically  produced  by  die  industrial  base. 
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3.6.3.  Outputs 

The  output  of  this  module  consists  of  the  number  of  each  type  of  system  that  can  be 
produced  during  a  given  time  period. 

3.6.4.  Issues 

Issue  1:  Do  the  variables  and  the  coefficients  used  in  the  optimization  fully 
accomplish  their  intended  effect? 

Background:  TTiere  are  two  approaches  to  use  when  making  a  cost-effectiveness 
evaluation[15].  They  are  a  fixed  effectiveness  approach  and  a  fixed  budget  approach. 

The  fixed  effectiveness  approach  has  a  specified  level  of  effectiveness  to  be  attain^  in 
the  accomplishment  of  a  given  objective.  This  approach  attempts  to  determine  which 
alternative  (or  combination  of  alternatives)  is  likely  to  achieve  the  specified  level  of 
effectiveness  at  lowest  economic  costs.  The  fixed  budget  approach  has  a  specified  cost 
level  to  be  used  in  attainment  of  a  given  objective.  This  approach  attempts  to  determine 
that  alternative  (or  combination  of  alternatives)  which  is  likely  to  produce  the  highest 
effectiveness.  The  overall  objective  in  either  approach  is  minimizing  costs  per  unit  of 
effectiveness  or  maximizing  effectiveness  per  unit  of  cost. 

Response:  The  VAA  methodology  uses  a  fixed  budget  approach  and  considers 
different  TOA  funding  levels,  each  of  which  is  evaluated  as  a  separate  optimization  run. 
Systems  that  appear  at  one  TOA  level  may  not  necessarily  appear  at  the  others.  Percent 
changes  in  combat  effectiveness  values  appear  to  be  minimal  reprdless  of  the  TOA 
level.  This  indicates  that  the  methodology  may  be  improved  if  is  were  based  on  a  fixed 
effectiveness  approach  rather  than  a  fixed  budget  approach.  The  objective  function  could 
be  changed  to  reflect  a  cost/  quantity  relationship  rather  than  an  effectiveness/  quantity 
relationship.  This  relationship  may  achieve  that  specified  level  of  effectiveness  at  the 
lowest  economic  cost. 

Issue  2:  Are  the  assumptions  in  this  module  specifically  stated  and  are  they 
reasonable? 

Response:  There  were  no  assumptions  made  for  this  module[12]. 
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4.  Validation  and  Verification 


4.1.  Summary 

The  overall  VAA  methodology  is  valid  for  assisting  decision  makers  in  evaluating  and 
comparing  competing  alternatives  in  the  POM  building  process.  The  verification  of  the 
methodology  must  be  an  ongoing  process  as  VAA  matures  and  expands  into  other  areas. 
VAA  is  capable  of  examining  tradeoffs  across  a  full  range  of  program  issues.  Future 
expansion  for  VAA  may  include  different  functional  or  mission  areas.  Trade-offs  of  this 
type  could  even  involve  such  areas  as  military  pay  vs  OMA  vs  RDA. 

The  methodology  uses  several  analytic  methods  to  measure  the  contribution  of  an 
alternative  to  the  current  program  base.  These  measurements  are  performed 
quantitatively  through  the  use  of  the  combat  model  CORBAN  and  the  costing  models 
CQM  and  LCCM.  The  measurements  are  performed  qualitatively  through  the  use  of  the 
AHP  and  Saaty’s  pairwise  comparison  technique.  The  TOPSIS  algorithm  combines 
these  two  measures  into  a  single  measiue  for  use  in  the  optimization.  The  combined 
measure  is  then  evaluated  against  cost  using  FOMOA. 

There  are  several  major  issues  that  exist  in  the  current  methodology  that  need  to  be 
addressed.  These  issues  involve  the  comparison,  scaling,  and  combination  of  different 
MOEs  and  the  assessment  framework  used  to  prioritize  among  compering  alternatives. 
These  issues  may  cause  rank  reversals  and  arbitrary  ranking  among  alternatives.  There 
are  other  issues  that  should  also  be  addressed.  They  involve  the  uncertainty  inherent  in 
the  cost  decision  process  and  the  fixed  budget  approach  performed  in  the  optimization. 

4.2.  Conclusions 

Several  enhancements  can  be  made  to  the  current  methodology  .  These  enhancements 
include  the  proposed  application  of  the  principles  of  MAUT  to  the  MOEs  generated  by 
CORBAN  in  the  explicit  effectiveness  module,  and  the  synthesis  of  MAUT  with  AHP  in 
the  implicit  effectiveness  module.  This  application  can  be  performed  through  the  use  of 
either  an  additive  or  multiplicative  multi-attribute  model.  This  would  eliminate  the  need 
for  the  TOPSIS  algorithm.  The  TOPSIS  algorithm  needs  to  be  eliminated  because  the 
output  from  TOPSIS  is  just  a  ranking  of  alternatives.  The  coefficients  forming  the 
objective  function  need  to  have  a  ratio  interpretation.  TOPSIS  considers  the 
decisionmaker  to  be  risk-neutral  and  this  may  not  be  the  case.  The  application  of  MAUT 
will  incoiporate  the  risk  preference  of  the  decisionmaker  into  the  decisionmaking 
process.  Other  enhancements  that  can  be  made  include:  incorporating  cost  uncertainty 
and  the  range  of  error  it  is  likely  to  produce  in  the  cost  estimate,adding  new  MOEs  which 
’incorporate  the  principles  of  AMand  Battle  Doctrine,  and  using  a  cost/quantity  approach 
rather  than  an  effectiveness/quantity  approach  in  the  optimization. 
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Appendix  A.  Analytic  method  used  at  TRADOC: 

TRADOC  analysts  use  a  concept  based  requirements  system  (CBRS),  AHP,  batdefield 
importance,  and  a  capital  allocation  algorithm  to  prioritize  modermzation  inMatives  m  - 
terms  of  their  level  of  importance  to  the  conduct  of  airland  operations[19]»  CBRS 
defines  how  the  Army  intends  to  fight  and  allows  TRADOC  to  plan  and  progr^ 
modernization  initiatives  in  a  given  cycle  of  time.  Battlefield  importance  is  related  to 
resource  value  and  is  expressed  in  terms  of  a  benefit-to-resource  ratio. 
allocation  algorithm  is  applied  to  their  prioritized  list  to  determine  overall  affordability. 
TRADOC  chose  the  AHP  and  Saaty' s  pairwise  comparison  process  as  Ae  ^alytical 
technique  for  prioritizing  the  benefits  associated  with  modernization  initiatives. 

The  CBRS  contributes  to  the  development  of  the  Army  Modernization  Memorandum 
(AMM)  and  the  Long  Range  Army  Materiel  Requirements  Plan  (LRAMRP).  The  AMM 
is  a  plan  for  improving  the  Army  in  the  areas  of  doctrine,  training,  leader  development, 
organization,  and  materiel.  The  AMM  prioritizes  recommendations  for  future  toy 
modernization.  AMM  94-08  prioritized  about  500  candidate  modernization  initiatives,  of 
which  400  were  materiel  initiatives  and  100  were  in  the  areas  of  doctrine,  leader 
development,  training,  and  organization.  The  LRAMRP  is  the  matenel  annex  m  the 
AMM  It  provides  ordered  year-by-year  RDA  funding  recommendations  to  HQDA  for 
the  management  decision  packages  (MDEPs).  MDEPs  are  developed  by  the  proponents 
in  conjunction  with  program  managers  and  are  the  subcomponents  of  the  AMM 
modernization  initiatives. 

The  AMM  development  process  is  hierarchically  structured.  The  first  level  is  the  force 
types  to  be  modernized,  which  is  examined  in  terms  of  conflict  intensity.  The  second 
level  is  the  battlefield  functional  mission  area  (BFMA)  elements.  The  third  level  is 
comprised  of  28  capability  packages  which  make  up  the  BFMA.  These  packages  are 
developed  jointly  by  TRADOC  schools  and  centers  (see  figure  A.l). 
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Figure  A.l.  AMM  Development  Process 


Once  the  packages  are  developed,  the  AMM  process  continues  in  three  stages.  In  the 
first  stage  evaluation  boards  are  convened  to  prioritize  modernization  actions  within  each 
capability  package.  The  boards  are  provided  with  information  regarding  capability 
package  definitions,  assessments,war  fighting  results,  analyses  examining  current  and 
ftiture  required  capabilities  for  selected  capability  packages,  and  results  of  analyses  of 
special  issues  (such  as  branch  planning  analyses).  Boards  use  the  pairwise  comparison 
process  to  assess  the  relative  importance  of  modernization  actions  and  establish  priority 
weights.  In  stage  two  a  senior  review  board  is  assembled  to  validate  the  resulting 
priorities.  The  validation  process  examines  where  the  emphasis  is  placed  within  and 
across  modernization  actions.  This  board  uses  the  same  pairwise  comparison  process  as 
the  evaluation  boards.  The  upper  level  of  the  hierarchy  (force  type  and  conflict  intensity) 
is  evaluated  by  senior  Army  commanders.  A  complete  list  is  then  assembled  by  merging 
the  results  of  stage  one,  stage  two,  and  the  goal  priorities  from  the  senior  commanders. 
Stage  three  compares  the  projected  cost  with  the  priority  list  using  their  capital 
allocation  software.  This  culminates  in  TRADOC's  input  into  the  Army-wide 
modernization  strategy. 
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Appendix  B.  Establishment  of  weights. 


The  impUcit  effectiveness  module  develops  the  hierarchical  weights  for  all  of 

of  the  ^sessment  hierarchy  and  individual  criteria.  The  assessment  hierarchy  consists  of 

three  levels  (see  figure  B.l). 


Figure  B.l  Effectiveness  Hierarchy 


Level  one  is  the  output  from  the  effectiveness  integration  module.  These  are  the 
effectiveness  coefficients  that  are  used  in  the  optimization  module.  Level  ^o  is  a 
hierarchical  breakdown  of  level  one  and  are  the  components  of  value  which  are  found  in 
the  explicit  and  implicit  effectiveness  modules.  The  dashed  lines  in  the  above  fi^ 
indicate  those  components  of  value  which  have  not  been  evaluated  at  this  time.  I^vel 
three  are  the  measures  of  effectiveness  generated  by  either  a  quanntattve  or  quahtattve 

model. 


The  establishment  of  weights  is  accomplished  by  making  pairwise  comparisons  within 
the  hierarchy  i.e,  comparing  the  elements  in  pairs  against  a  set  criteria.  An  example  of 
such  a  comparison  is  shown  in  figure  B.2  : 
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Asset  Versatility 
and  Deployability 

Asset  versatility  and 
deployability  measures  the 
applicability  of  a  program  to 
muHipie  theaters  of  operation. 


Political  Risk: 
vs.  Public  Opinion 

The  public  opinion  measure  of 
political  risk  Is  a  sublective 
evaluation  of  the  attitudes  of 
the  general  public  toward  a 
proposed  program. 


V®ry  Vary 

Absolute  Strong  Strong  Week  Equal  Weak  Strong  Strong  Abaolute 

□□□□□□□□□□□□□□□□□ 

Chock  tho  box  which  you  bolleve  Is  presently  the  best  measure  of  the 
_ Importance  of  the  Hem  on  the  left  as  compared  to  the  Hern  on  the  right. 


Figure  B.2.  Sample  Pairwise  Comparison 


The  relative  importance  the  decisionmaker  assigns  to  each  of  the  elements  during  the 
comparison  is  then  translated  to  a  1  -  9  scale.  The  translated  numbers  and  their  inverses 
are  placed  into  a  decision  matrix  and  the  geometric  mean  method  is  used  to  determine  the 
weights.  The  w’eights  for  all  MOEs  under  each  component  of  value  must  sum  to  one. 


Pairwise  comparisons  are  also  made  between  the  components  of  value  i.e,  combat 
effectiveness  vs.  SIAM  factors.  Weights  are  obtained  for  each  of  the  components  of 
value  and  together  they  also  sum  to  one. 
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Appendix  C.  Secondary  Impact  Analysis  Modifiers 
(SIAM)  Factors. 

SIAM  factors  arc  key  factors  which  are  ^nside^d  in  the  decisionmaking  process,  bat 
are  not  official  Army  measures  of  effectiveness  . 

production  requirements. 

^£'‘^:“an'fp“s^ peat  forL  pmgram  to  significant  opposition  » the 
program. 

a  Political  Risk-  Congressional  Opinion.  The  congressional  opinion  measure  of 

wl?  n?^k  tea  suWeetive  evaluan^n  of  Ihe  attitudes  of  the  Congress  towards  a 
?^^^"pre^r  «  fiom  significant  positive  support  for  the  program  to 

significant  opposition  to  the  program. 

4.  POUdcalRisk:  B-^^^rre^onhtS 

Chiefs  of  Staff  toward  aproposed 

program.  It  ranges  from  significant  positive  support  for  the  program  to  sigrafican 
Opposition  to  the  program. 

Political  Risk*  Internal  Army  Opinion.  The  internal  Army  opinion  of  political  risk  « 
a  sScSt  elSnariZf  the  atLdes  of  special  interest  gmups  Mthm  th' 
a  proposed  program.  It  ranges  from  significant  positive  support  for  the  p  gr 

significant  opposition  to  the  program. 

^■o^t^'^Sfngefs  measL^^^^ 

^Ss  of  the  Army.  Change  is  subjectively  measured  on  a  scale  ranging  from  extreme 
change  to  marginal  change. 

7  ProeramHexibUity.  Ameasnreof  the  funding  flexibility  associated  with  a  pro^. 

M™rrth:;"%or"  a  a  sl^rfic:?t  penalty 

clause  would  have  low  flexibility. 


4CAA  Study  Report  91-9  p.7-3 
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8.  Program  Feasibility.  A  subjective  measure  of  the  number  of  obstacles  which  might 
prevent  program  execution.  Programs  which  are  extremely  complex  or  slow  to 
implement  have  negative  ratings  for  feasibility,  while  programs  with  few  bureaucratic 
hurdles  have  positive  ratings  for  feasibility. 

9.  Asset  Versatility  and  Deployability.  Asset  versatility  and  deployability  measures  the 
applicability  of  a  program  to  multiple  theaters  of  operation.  For  example,  a  propam  that 
bought  new  weapons  that  could  be  used  in  all  theaters  would  have  a  relatively  high  rating 
for  asset  versatility. 

10.  Operational  and  Technical  Risk.  Operational  and  technical  risk  is  a  subjective 
measure  of  the  probability  associated  with  a  program  meeting  all  of  its  stated 
performance  criteria.  For  materiel  systems,  this  factor  would  roughly  correspond  to  the 
stage  of  materiel  development  for  the  system.  For  example,  a  system  in  stage  6.1  will 
have  a  higher  risk  than  a  system  in  stage  6.3. 
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Appendix  D.  Glossary. 

Abbreviations,  Acronyms,  and  Short  Terras: 


ACP 

AHP 

AMC 

ARIM 

AMM 

AMVF 

BDP 

BFMA 

BPS 

BCE 

CAA 

CBRS 

CEAC 

CER 

COFM 

CORBAN 

CQM 

CSA 

DA 

DCSOPS 

DPAE 

EBR 


Army  cost  position 

analytic  hierarchy  process 

US  Army  Materiel  Command 

Army  Resource  Integration  and  Management  Study 

Army  Modernization  Memorandum 

additive  measurable  value  function 

battlefield  development  plan 

battlefield  functional  mission  areas 

blue  force  surviving 

baseline  cost  estimate 

Concepts  Analysis  Agency 

concept  based  requirements  system 

US  Army  Cost  and  Economic  Analysis  Center 

cost  estimating  relationship 

correlation  of  forces  and  means 

Corps  Battle  Analyzer 

Cost  Quantity  Model 

Chief  of  Staff  of  the  Army 

Department  of  the  Army 

Deputy  Chief  of  Staff  for  Operations 

Director,  Program  Analysis  and  Evaluation 

effective  battalions  remaining 
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FER 

force  exchange  ratio 

FOMOA 

Force  Modernization  Analyzer 

FR 

force  ratio 

FY 

fiscal  year 

ISC 

Information  Systems  Command 

LER 

loss  exchange  ratio 

LCCM 

Life  Cycle  Cost  Model 

LP 

linear  program 

LRRDAP 

Long-Range  Research,  Development  and  Acquisition  Plan 

LRAMRP 

Long-Range  Army  Materiel  Requirements  Plan 

MACOM 

major  Army  command 

MAU 

multi-attribute  utility 

MAUT 

multi-attribute  utility  theory 

MCA 

military  construction,  Army 

MOE 

measure(s)  of  effectiveness 

OMA 

operation  and  maintenance.  Army 

OPLAN 

operation  plan 

POM 

Program  Objective  Memorandum 

PPBES 

Program,  Planning,  Budgeting  and  Execution  System 

RAM 

reliability,  availability,  and  maintainability 

RDA 

research,  development,  and  acquisition 

RDTE 

research,  developing,  testing,  and  evaluation 

RSM 

response  surface  methodology 
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SER 

systems  exchange  ratio 

SIAM 

secondary  impact  analysis  modifiers 

TOA 

total  obligation  authority 

TOPSIS 

Technique  for  Order  PrefeKjnce  by  Similarity  to  Ideal  Solution 

TRADOC 

US  Army  Training  and  Doctrine  Command 

UT 

utility  theory 

VAA 

Value  Added  Analysis 
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Appendix  E.  Briefing  Slides 


Op<nli«u  Kwwrrii  Ctaler 
UkiUri  Stale*  MiEtary 
W«*tPfEBl.Nc»Y«rt  Ifm 


Validation  and  Verification 
of  the  Army  Program 
Value  Added  Analysis 


Proi*ct 

Sponsor 

Analyst 


Program  Analysis  and  Evaluation 
CPT  Michael  G.Titone 


"  ■■  Essence  of  VA A 
VAA  is  a  way  of  bringing  together 
all  of  the  different  constraints  that 
impact  acquisition  strategy.  It 
defines  the  feasible  region  for 
decision  makers. 


To  provide  a  timely  end  unbiased  analysis  of  the 

Army  Program  Value  Added  Analysis  and  make 
recommendations  for  desirable  modifications  to 

Validation  -  Building  the  right  model. 

•lha  proeau  of  datarmlning  tha  axtarrt  to  which  modalt  and 
obnulatlons  aceorataly  rapraoant  tha  raal  world  from  tho  parapocthra 
of  tha  tntandad  uaa  of  tha  mod  at. 

m 

the  methodology. 

p.  PA&E 

Verification  -  Building  the  model  right. 

*  tha  procMS  of  datarmlnkig  that  modalo  and  tlmulalloni  accurataty 
rapraaant  tha  davalopar'a  concaptual  daacrtption  and  apactficatlona. 

'Citation  of  Related  Work* 


•r,  Jack  U.  dr. 

,,  MB)pr,USA 
^j!  A  CrWcat  Analyils  of  **Tho 
1  Tochniquo  for  Ordof 
I  PfolmioobySImitartty  to 
}  tdool  Solufloft**  TOPSS. 
i  Wool  Pokrt,  Hi:  R*p©rt 
g  NumbofSM.  duly  1991 

OARCOM-P  709-102 
^  Enfilnoortng  Ooaipii 
Hofidbook  Army  Woopofi 
Syotom  Aftolyolo  Part  I, 
Oetobar  1979. 

Motor  John  (1974)  I 
Unoor  Stadatlcal  Modolo 


i|%/ndoroott,  Miehoot 
9RACOAO  CAAO 
Brloltng  Noloo  on  ** 
Doctalon  Sopport  for 
Anny  Modorriliotton**,  21 
April  1992 

y.  Ralph  L  (1979) 
OoctaloRi  whh  MuMpte 
Ob)octtvoo 

Oyor  Jareot  S.'Ttamaffca 
on  mo  Analytic 
Htororchy  Piocooo*; 


t  Oporatlons  Roooareh  Cantar 


I  EKimpia:  Ropfooomadvo ArmoWAntl-A^Of_Sy«1onm^ 


taaua:  Givan  kmrtad  dollar  roaoureo*.  wOtich  mi  of  A3 
Systama  should  ba  procuiod? 


AtttffiitlYtl 

Charictaflitlca 

Basocoso 

Europa  Soanano  with  2S  tyatama 

Aft  1 

M1A2 

NLOS-AT 

MLRS  SAOAPM 

A»2 

MtA2 

AMS-H 

BRAOCEY  BLK II 

MLRS  SADARM 

I  Oparatlont  Raoaareh  Cantar 
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■■  Validation  — — 

pvrall  Methodology  *  R«spons«t 

An  attomativt  Is  m#tsur*d  both 
•xpllcltly  and  implicitly. 

IMUM  MMdatftd  wWl  AHP 
Imum  aMOclBttd  with  MO€s 
tmum  aM0cM«4  wtth  acaing 

Valuta  art  eomblntd  into  a  singit 
mtaaurt  of  btntfit.  That  mtaaurt  la 
optimizad  againat  cost 

iMUM  aMOclatad  wMi  TOPSIS 


[  Opantient  flMMrch  Caotar 


. Verification  — — 

Issue  Clarification  -  Major  Issuts 

What  art  tht  ramifications  of  discarding 
certain  alttmativts  whan  tvaluating 
multipit  systems  In  various  combinations? 

Who  decides  on  Issuts  being  kept  or 
discarded? 

How  do  Issues  drive  the  experimental 
design  matrix? 


Oparatkona  Paaaarch  Cantar 


'  Verification 

Issue  Clarification  -  Responses 


Response  surface  methodology  is  used  to 
^  take  specific  comblrwtions  of  the  total 
combinations  possible. 


Sponsors  arxl  VAA  analysts  decide  the 
issues.  Selection  of  issues  is  based 
on  the  importance  of  the  issue  and  the 
ability  to  model  the  systems  Involved. 


Issues  are  framed  by  the  selection  of 
the  systems  and  the  year/scenario 
combinations  to  be  played. 


>  Oparaliona  Raaaarch  Cantar 
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fipsttrifl  -  Issues 

Costing  «  Response 

-  Production  cost  elements  are  the  most 

Can  the  derivation  of  the  most 

sensitive.  The  CQM  end  LCCM 

sensitive  cost  elements  be  Improved 

estimate  the  Impact  of  production 

without  significant  computational 

variations  on  life  cycle  costelemenU 

burden? 

and  appropriations.  These  methods 

need  no  Improvement. 

Who  obtains  the  Information  from  the 

Baseline  Cost  Estimate  (BCE)  and 

CAA  collects  ail  cost  data  In  coordination 

Army  Cost  Position  (ACP)  and  how 

with  CEAC.  BCE  and  ACP  data  come  from 

accurate  are  they  for  life  cycle  cost 
data? 

Program  Managers. 

'  '  Verification 

Costing  -  Enhancements 


The  overali  cost  decision  process  could 
be  Improved  If  cost  uncertainties,  and  the 
range  of  error  they  are  likely  to  Introduce, 
are  incorporated  In  the  cost  estimate. 


I  operations  Beiosreh  Csntsr  J 


■Verification 

gYPilelt  Effectiveness  -  Ma|or  Issues 


. .  Verification 

^Ypllcit  Effectiveness  -  Response 


What  are  the  problems  associated  with 
the  Measures  of  Effectiveness  (MOEs) 
generated  by  the  combat  model? 


MOEs  are  too  dependent  end  similar  In 
nature.  MOEs  should  support  the  type 
of  Army  you  want  to  build  based  on  the 
Airland  Battle  tenets. 


Are  there  MOEs  that  give  better  Insights? 


t  Oporations  Rossarch  Cantor 


The  MOE,  toss  Exchange  Ratio,  Is 
redundant  and  can  be  eliminated.  MOEs 
Incorporated  into  the  combat  model  could 
be  the  percentage  of  blue  force  casualties, 
and  combat  system  utilization. 

Oporatlons  naaaarch  Cantor 
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•Verification’ 


gxpllcit  Effftctivness  •  Enhancements 


Multiattribute  utility  theory  can  be 
used  to  develop  DM  preference 
functions  for  each  MOE.  MOEs  could 
then  be  scaled  and  effectively 
combined  among  alternatives. 


^  Op*rit)ofis  RmmtcA  Ciint*r 


•Verification* 


Issue  Clarification 
Module 


i 


Explicit 

Cost 

Effectiveness 

Module 

Module 

— 1 

Effectiveness 

Integration 

Module 


I  Optimization 

Mnduitt. 


o<^i< 


I  Op*rMlofis  Witareh  OMrt»r  ■! 


•Verification' 


Implicit  Effeetlv>n»8«  •  Rcspont* 


The  possibility  exists  that  rank  reversal 
arid  arbitrary  ranking  may  occur  with 
the  use  of  AHP  and  Saaty*s  approach. 


AHP  is  a  good  techniques  for  obtaining 
the  weights  In  the  hierarchy.  Alternatives 
being  evaluated  need  to  be  rescaled. 


^  Op«ratlont  RMoareh  Cantar 


•Verification* 


Implicit  Effectiveness  -  Enhancements 

The  synthesis  of  MADT  with  the  AHP 
will  eliminate  the  possibility  of  rank 
reversal  arxi  arbitrary  ranking  among 
alternatives. 


Application  of  MAUT  will  provide  a 
frame  of  reference  and  an  appropriate 
scale  of  measure  to  combine  all 
Implicit  arxi  explicit  MOEs. 


i  Opanelona  Raiaarch  Cairtar 


•Verification’ 


Issue  Clarification 
Module 


i 


Explicit 

Implicit 

Effectiveness 

Module 

pk _ 

Cost 

EffectI  veness 

Module 

WiOQUIV 

1 

Optimizadoit 

Module 

r 

I  Oparatlona  Haaaarch  Cantar  i 
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•Effectiveness  Hierarchy  ■ 


Vaiu*  Added 
Effeettveness 
Coefficients 


Level  1.0 


EffectIveneM 


"T^vel  2.0 


[iol^QiMltty  jpOlherAnny  JseoondsiY 


j  efUfe _ II  Objectives  j  Analysis  WodWIers 


Level 


I  Operations  Research  Center  J 


•Verification* 


Effectiveness  Intearatfon  -  Major  Issue 


Does  theTOPSIS  methodology  reflect 
•  suitable  and  intrinsically  meaningful 
process  for  the  generation  of  value 
coefficients  for  use  in  the  optimization? 


>  Qperattons  ReeeereH  Center  ^ 


Effectiveness  Inteoraton  -  Response 

Effeetivenesa  Integration  •  Enhancements 

The  TOPSIS  output  consists  of 
coefficients  that  are  based  on  an 
ordinal  scale.  They  need  to  be  on  a 
ratio  scale  for  use  in  the  optimization. 

The  synthesis  of  MAUT  with  AHP 
eliminates  the  possibility  of  rank 
reversal  and  arbitrary  ranking  among 
alternatives  that  may  occur  using 

TOPSIS. 

TOPSIS  may  have  a  problem  with  rank 
reversal  and  arbitrary  ranking. 

The  AMVFfrom  MAUT  allows  MOEsto 
be  directly  combined  using  either  an 
additive  approach  or  a  multiplicative 
approach  and  The  TOPSIS  algorithm 
does  not  need  to  be  used. 

Lll  — — 
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Optimixatlon  -  Rasponss 

ODt?ml?fltIon  •  EnhencemenU 

Thamalfwdologyutasaflxadbiidgat 
epproech  In  maldng  their  cost 
effectiveneee  evaluation. 

Percent  changes  In  combat  affactivanasa 

. '  appear  to  be  mlnimsi  regardless  of  tha  TOA 

level.  This  Indicates  a  fixed  affactivanasa 
approach  should  be  conaldarad. 

Th#  objaetlv#  function  should  be 
changed  to  raflact  a  cost/ quantity 
ralationship  rathar  than  an 
affaetlvanass/  quantity  ralationship. 

.  Summary  — — — 

7h»r»  w  MVtrat  malor  Issues  that  exist  with  the 
current  methodology  that  need  to  be  resolved. 
These  Issues  Involve  the  comparison,  scaling,  and 
combination  of  different  MOEs  and  the  assessment 
franwwork  used  for  prioritization.  These  Issues 
can  cause  rank  reversal  and  arbitrary  ranking. 


The  application  of  MAUT  should  resolve  the  Issues. 
This  application  can  be  performed  using  either  a 
linear  or  multiplicative  model. 


>  Op«Mtlont  a«tMrcli  » 


Conclusions 


Th.  owrall  m.lhodolegy  tt  valid 
for  tscisttng  dMlsion  makars  in 
avaluatlng  and  comparing  compaUng 
altamallvat  In  tha  POM  building  procass. 


The  varlficaUon  of  tha  methodology  must 
be  an  ongoing  process  as  VAA  matures 
and  expands  into  other  areas. 


Of>«rallont  RMoarch  CtRl*r 
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